A comparison study of IRT calibration methods for mixed-format tests in vertical scaling

The purpose of this dissertation was to investigate how different Item Response Theory (IRT)-based calibration methods affect student achievement growth pattern recovery. Ninety-six vertical scales (4 × 2 × 2 × 2 ×3) were constructed using different combinations of IRT calibration methods (separate, pair-wise concurrent, semiconcurrent, and concurrent), lengths of common-item set (10 vs. 20 common items), types of common-item set (dichotomous-only vs. mixed-format), and numbers of polytomous items (6 vs. 12) for three simulated datasets differing in the number of examinees sampled per grade (500, 1000, 5000). Three indexes (absolute bias, standard error of equating and root mean square error) were used to evaluate the performance of the calibration methods on proficiency score distribution recovery over 80 replications. These indexes were derived for seven growth distribution criterion parameters (mean, standard deviation, effect size, and proportions of examines within four proficiency categories). Although exceptions were found in the results for all criterion parameters, important general trends did emerge. Pair-wise concurrent and semi-concurrent calibration methods performed better than concurrent and separate calibration methods for most criterion parameters and combinations of research conditions. Separate calibration, the vertical scaling method used most often in practice, provided the poorest results in most instances. Accuracy of vertical scaling also typically improved with larger samplings of examinees, more common items, mixing item formats in the common-item set, and increases in the number of polytomous items in the common-item set. General trends and exceptional cases from the various analyses are described in